Papers with dual encoder
Retrieval-Enhanced Dual Encoder Training for Product Matching (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Recent work has proposed a dual encoder for product matching due to its high performance and computation efficiency. |
| Approach: | They propose retrieval-enhanced dual encoder training to improve product matching . they use public and real-world product matching datasets to train the dual encoded model . |
| Outcome: | The proposed approach improves on a public and real-world product matching datasets. |
Efficient Nearest Neighbor Search for Cross-Encoder Models using Matrix Factorization (2022.emnlp-main)
Copied to clipboard
| Challenge: | Efficient k-nearest neighbor search is a fundamental task, foundational for many problems in NLP. |
| Approach: | They propose an approach that avoids the use of a dual-encoder for retrieval, relying solely on the cross-encoding model. |
| Outcome: | Empirically, for k > 10, our approach provides test-time recall-vs-computational cost trade-offs superior to the current widely-used methods that re-rank items retrieved using a dual-encoder or TF-IDF. |
NAIL: Lexical Retrieval Indices with Efficient Non-Autoregressive Decoders (2023.emnlp-main)
Copied to clipboard
| Challenge: | Neural document rerankers require dedicated hardware for serving, which is costly and often not feasible. |
| Approach: | They propose a method that captures 86% of the gains of a Transformer cross-attention model with a lexicalized scoring function that only requires 10-6% of . the model architecture is compatible with recent encoder-decoder and decoder-only large language models, such as T5, GPT-3 and PaLM. |
| Outcome: | The proposed model captures 86% of the gains of a Transformer cross-attention model with a lexicalized scoring function. |
ConveRT: Efficient and Accurate Conversational Representations from Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | ConveRT is a pretraining framework for conversational AI that is computationally heavy, slow, and expensive to train. |
| Approach: | They propose a pretraining framework for conversational tasks that is efficient, lightweight, and inexpensive. |
| Outcome: | The proposed model achieves state-of-the-art performance across widely established responses . it trains substantially faster than existing state- of-the art models . |
MURAL: Multimodal, Multitask Representations Across Languages (2021.findings-emnlp)
Copied to clipboard
Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, Jason Baldridge
| Challenge: | Image-caption pairs and translation pairs provide the means to learn deep representations of and connections between languages. |
| Approach: | They propose a dual encoder that integrates image-text matching and translation pairs to solve two tasks by learning from billions of pairs. |
| Outcome: | The proposed encoder outperforms ALIGN's cross-modal retrieval performance on well-resourced languages and significantly improves on under-resource languages. |
Preserving Zero-shot Capability in Supervised Fine-tuning for Multi-label Text Classification (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods that assume label descriptions ensure zero-shot capability lose their zero-shot capability during training. |
| Approach: | They propose a method that preserves the zero-shot capabilities of powerful dual encoders and label-wise attention networks by freezing the label encoder. |
| Outcome: | The proposed methods preserve the zero-shot capabilities of powerful dual encoder and label-wise attention network architectures by freezing the label encoder. |
Fusion or Defusion? Flexible Vision-and-Language Pre-Training (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to vision-and-language pretraining (VLP) lack effectiveness and efficiency in downstream multimodal tasks. |
| Approach: | They propose a flexible vision-and-language pre-training model by incorporating cross-modal fusions into a dual-encoder architecture and a cross-module knowledge transfer strategy to guide the training process. |
| Outcome: | The proposed model is well-equipped with effectiveness and efficiency compared with other strong VLP models. |
Multi-Vector Attention Models for Deep Re-ranking (2021.emnlp-main)
Copied to clipboard
| Challenge: | Document retrieval systems often use two styles of neural network models . dual encoder models are used for retrieval and deep re-ranking, while cross-attention models are typically used for shallow reranking. |
| Approach: | They propose a dual encoder and cross-attention neural network architectures that combine query and document representations to optimize retrieval accuracy. |
| Outcome: | The proposed architecture trades off retrieval accuracy with joint computation and offline document storage cost. |
Exploring the Synergy of Dual-path Encoder and Alignment Module for Better Graph-to-Text Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | KG-to-text generation model lacks explicit graph-text alignment strategy due to discrepancy between textual and structure information. |
| Approach: | They propose a synergetic knowledge graph-to-text model with a dual-path encoder, alignment module and guidance module to solve these problems. |
| Outcome: | The proposed model achieves competitive performance on three benchmark datasets. |
Entity Linking in 100 Languages (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to multilingual entity linking are cross-lingual, with a focus on zero-shot evaluation. |
| Approach: | They propose a new formulation for multilingual entity linking where language-specific mentions resolve to a language-agnostic Knowledge Base. |
| Outcome: | The proposed model outperforms state-of-the-art models on a large multilingual dataset and shows that frequency-based analysis provided key insights for the model and training enhancements. |
Large Dual Encoders Are Generalizable Retrievers (2022.emnlp-main)
Copied to clipboard
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, Yinfei Yang
| Challenge: | Experimental results show that dual encoders outperform sparse and dense retrievers on the BEIR dataset significantly. |
| Approach: | They challenge belief that bottleneck layer is too limited for out-of-domain generalization . they scale up the model while keeping bottleneck as a single dot-product with a fixed size . |
| Outcome: | The proposed model outperforms sparse and dense retrievers on the BEIR dataset significantly. |
Investigating Multi-layer Representations for Dense Passage Retrieval (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Dense retrieval models adopt vectors from the last hidden layer of the document encoder to represent a document, which is in contrast to the fact that representations in different layers of a pre-trained language model contain different kinds of linguistic knowledge and behave differently during fine-tuning. |
| Approach: | They propose to utilize representations from multiple encoder layers to make up the representation of a document, which they denote Multi-layer Representations (MLR). |
| Outcome: | The proposed model outperforms dual encoder, ME-BERT and ColBERT in the single-vector retrieval setting and with other advanced training techniques. |